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ABSTRACT 

The MiniBooNE results have still not been able to comprehensively rule out the oscillation inter- 
pretation of the LSND experiment. So far the so-called short baseline experiments with energy 
in the MeV range and baseline of few meters have been probing the existence of sterile neutrinos. 
We show how signatures of these extra sterile states could be obtained in TeV energy range at- 
mospheric neutrinos travelling distances of thousands of kilometers. Atmospheric neutrinos in the 
TeV range would be detected by the upcoming neutrino telescopes. Of course vacuum oscillations 
of these neutrinos would be very small. However, we show that resonant matter effects inside 
the Earth could enhance these very tiny oscillations into near-maximal transitions, which should 
be hard to miss. We show that imprint of sterile neutrinos could be unambiguously obtained 
in this high energy atmospheric neutrino event sample. Not only would neutrino telescopes tell 
the presence of sterile neutrinos, it should also be possible for them to distinguish between the 
different possible mass and mixing scenarios with additional sterile states. 
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1 Introduction 



Last few years have seen a tremendous progress in the field of Neutrino Physics, so much so that 
its fair to say that neutrinos have now become a pivot in our understanding of physics beyond 
the standard model of particle physics. The first conclusive prove of the existence of neutrino 
mass and mixing came from the observation of atmospheric neutrinos by the SuperKamiokande 
detector [1]. The zenith angle dependent data on atmospheric neutrinos from this experiment 
could be explained only if neutrinos oscillate with Am^ ~ 2.1 x 10~ 3 eV 2 and sin 2 2# 23 = 1. 
This was followed by the spectacular results on solar neutrinos from the SNO experiment, which 
proved beyond doubt that solar neutrinos do indeed oscillate, corroborating the observations of 
all earlier solar neutrino experiments, the Homestake, SAGE, GALLEX/GNO, Kamiokande and 
SuperKamiokande [2]. The so-called Large Mixing Angle (LMA) solution emerged as the only 
solution of the solar neutrino deficit problem with Am^ = 6 x 10~ 5 eV 2 and sin 2 9 12 = 0.31 [3]. 
The LMA solution was confirmed by the KamLAND reactor antineutrino experiment [4] and the 
combined solar and KamLAND data choose Ara^ = 8 x 1CT 5 eV 2 and sin 2 9 i2 = 0.31 as the best- 
fit parameter values [3]. The SuperKamiokande atmospheric neutrino results were affirmed by two 
terrestrial accelerator-based experiments - K2K [5] and MINOS [6] and the combined atmospheric 
and accelerator data demand Am^ = 2.4 x 10~ 3 eV 2 and sin 2 2# 2 3 = 1 [3]. Another very important 
result we have is from the reactor antineutrino experiment Chooz [7]. Results of this experiment, 
analyzed along with the other neutrino data impose the constraint that sin 2 #i 3 < 0.04 at 3<r C.L. 
[3]. " 

The latest addition to the repertoire of experimental result on neutrinos comes from the Mini- 
BooNE experiment [8]. The MiniBooNE experiment was set-up to reconfirm the positive oscilla- 
tion signal reported by the LSND collaboration [9], which so-far remains the only experiment to 
have seen neutrino oscillations at a frequency which demands Am 2 in the eV 2 range. All other 
short baseline experiments [10] have been consistent with the hypothesis of null oscillations. The 
MiniBooNE experiment also reported to have not seen oscillations in the energy regime consistent 
with LSND. The extra mass squared difference demanded by the LSND signal can be accommo- 
dated along with solar and atmospheric neutrino results only if there were sterile neutrinos. The 
most economical scenario comes from adding just one extra sterile neutrino, giving the so-called 
2+2 and 3+1 neutrino mass schemes [11]. It has been shown that the 2+2 spectrum is strongly 
disfavored from the solar and atmospheric neutrino data. The 3+1 scheme on the other hand 
suffers from a strong tension between the positive signal at LSND and null signal in all other short 
baseline experiments. The addition of the MiniBooNE results puts even stronger constraints on 
the 3+1 picture, disfavoring it at a very high C.L. [12]. Adding two extra sterile neutrino would 
give us the so-called 3+2 neutrino mass spectrum [13, 14, 15]. This picture interestingly gives 
a reasonable explanation of all neutrino oscillation data including LSND and MiniBooNE, if CP 
violation is allowed [12]. In [16] the author find a very good fit to world neutrino data for a CP 
conserving 3+2 mass spectrum as well. 

The situation concerning sterile neutrinos therefore seems to be far from settled. MiniBooNE 
was especially designed to confirm or refute the LSND signal and they have reported to have 
contradicted the LSND claim of positive oscillation signal. However, their first data set is with 
neutrinos while LSND had seen oscillations of antineutrinos. In addition the entire event sample 
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of MiniBooNE is not yet fully understood. They have seen excess electron events in their low 
energy sample, which still remains unexplained. It is hoped that this systematic excess of electron 
events seen in the experiment will eventually be explained. MiniBooNE is also now running in the 
antineutrino channel and results from this data set might settle the issue regarding the mismatch 
between the LSND and MiniBooNE results. 

Resolution of this perplexing issue could also come from other kind of experiments. Presence 
of sterile neutrinos would lead to distinctive features in the resultant supernova neutrino signal in 
terrestrial detectors such as future megaton water Cerenkov detectors and neutrino telescopes like 
IceCube [14]. Very recently it has been shown that the planned and up-coming next generation 
reactor neutrino experiments such as Double Chooz, Daya Bay, Angra, and RENO, which are 
being built to probe the mixing angle #13, could also check for the existence of sterile neutrinos 
through combination of data from the near and far detectors [17]. Not only should it be possible to 
cross-check the 3+2 neutrino mass scheme at these experiments, we should also be able put limits 
on the mixing angles involving sterile neutrinos [17]. Possibility of observing sterile neutrinos in 
the upcoming accelerator-based long baseline experiments was studied very recently in [18] for 
the conventional CNGS experiment, and in [19] for the future neutrino factory. In another recent 
paper we have shown that the existence of sterile neutrinos could in principle also be probed in 
the ultra high energy neutrino signal in the neutrino telescopes [20]. 

In this paper we show how the data from very high energy atmospheric neutrinos in neutrino 
telescopes such as IceCube [21], Km3Net [22], NEMO [23] and NESTOR [24] could be used to check 
if sterile neutrinos indeed exist. The neutrino telescopes will have energy threshold of about 100 
GeV and are designed to observe ultra high energy neutrinos. The very high energy atmospheric 
neutrinos with energy in the range 10" 1 — 10 4 TeV will also be observed in these detectors and they 
will in principle constitute the "background" for the ultra high energy neutrino "signal" . However, 
this atmospheric neutrino "background" in km 3 neutrino telescopes will be sizable and can hence 
be used to provide crucial information on some physics issues. The AMANDA experiment has 
already observed the high energy atmospheric neutrinos, and the observed flux is reported to be 
consistent with the theoretical predictions [25]. In 10 years of operation, IceCube will be able to 
collect 7x 10 5 atmospheric muon neutrino events [26]. With such a huge data sample, it was shown 
in [26] that the atmospheric neutrino events in IceCube could be used to put severe constraints on 
non-standard physics. Feasibility studies of constraining non-standard physics in ANTARES was 
performed in [27]. We will show that for the mass squared difference needed to explain LSND, 
we expect near-resonant matter effects between the active and sterile neutrino states inside the 
Earth. This leads to drastic changes in the expected flux at the detector. This change is both 
energy as well as zenith angle dependent and should provide foolproof signal for the existence of 
sterile neutrinos in this mass regime. 

The paper is organized as follows. In Section 2 we discuss the enhancement of neutrino mixing 
and oscillations from the matter effects due to the extra sterile states. The numerically calculated 
exact oscillation probabilities for the PREM matter density profile of the Earth is presented in 
Section 3 for the simpler (though disfavored) 3+1 spectrum. We reiterate that we present results 
for this case purely for illustration only. With the 3+1 exemplary case, we hope to highlight some 
of the features of the active-sterile resonant matter effects. In section 4 we show the oscillation 
probabilities for the more realistic 3+2 neutrino mass spectrum. We make some comments in Sec- 
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tion 5 on the detection of the atmospheric neutrinos in neutrino telescopes and possible signatures 
of sterile neutrinos in the data sample. We end in Section 6 with discussions and conclusions. 



2 Neutrino Mixing in Matter with Sterile Neutrinos 

Neutrinos would undergo maximum flavor conversion in vacuum when the oscillatory term 

, /Am 2 ii L\ t s 

sin (-if-) = 1 • w 

where L and E are the distance travelled by the neutrinos and Amjj = m 2 — mf. This happens 
when their energy corresponds to the value 

E (TeV) = 0.81 x 10~ 3 ^AmJ (eV 2 )) (l (km)) , 

= 81 x (r$) x (laWto) ■ (2) 

Thus we see that if sterile neutrinos are mixed with the active ones with Am-j ~ eV 2 , we expect 
to see maximum flavor conversions for neutrinos with energies in the range of a few TeV, if 
they are traveling over distances in the range of 10,000 km. High energy atmospheric neutrinos 
travel distances of this order to reach the neutrino telescopes. They would therefore encounter 
these flavor oscillations. The amplitude and hence the extent of the oscillations is of course 
determined by the corresponding mixing angle, which for sterile neutrinos are usually constrained 
to be extremely small 1 . The combined errors coming from the uncertainties in the predicted high 
energy atmospheric neutrino fluxes and the experimental uncertainties, could threaten to wash 
out these oscillations driven by tiny mixing angles. 

However, atmospheric neutrino travel through the matter before they reach the detector and 
this could produce drastic changes in the amplitude of the active-sterile oscillations 2 . In fact, the 
focal point of this paper is the very large matter effects which the neutrinos pick as they move 
inside the Earth's matter. In presence of matter the neutrino mass squared matrix changes to 
[31, 32, 33] 

M 2 F = UMtf + A (3) 
where U is the unitary mixing matrix relating the mass eigenstates to the flavor eigenstates, 

M. = Diag{m\,m\,rn\,m 2 A ) or Diag{m\ ) m\ ) m\ ) m\ ) m\) , (4) 

A = Diag(A cc , 0, 0, A NC ) or Diag(A cc , 0, 0, A NC , A NC ) , (5) 



1 See [20] for consequences of large active-sterile mixing for ultra high energy neutrino signal in the neutrino 
telescopes. 

2 Matter effects for very high energy neutrinos due to presence of sterile neutrinos has been discussed before in 
some form in [28, 29, 30]. 
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for the 3+1 and 3+2 neutrino mass spectrum respectively, where 



ice* 



±2\f2G F pN A Y e E , (6) 



A NC = ±V2G F pN A (l - Y e )E . (7) 

Here the quantities A C c and — A NC are the matter induced charged current and neutral current 
potentials respectively, and given in terms of the Fermi constant Gp, matter density p, Avagadro 
number N A , electron fraction Y e and energy of the neutrino E. The "+" ("— ") sign in Eqs. (6) 
and (7) corresponds to neutrinos (antineutrinos). In the above equations we have re-casted the 
mass matrix in such a way that the neutral current component —A^c, which is negative and which 
appears for all the three active flavors, is filtered out from the first three diagonal terms and hence 
it stays back as positive A^c for the sterile state(s), since they do not have any weak interactions. 
Presence of matter dependent terms in the mass matrix modify the mass squared differences and 
mixing angles of the neutrinos in matter and these quantities are given respectively as 



(Am%) M = v /(Am] l cos2%-A M )2 + (Am^sin2%)2, (8) 

Am 2 • 

sin26# = sin2%^-^. (9) 

In the above we have assumed that only two neutrino states are predominantly involved and 

depending on which neutrino states these are, Am could be Acc, A^c or Acc ~ A^c- In this 

approximation the condition for v e — > or v e — > v T resonant transition when sin 2 20fj = 1, is 
given by 



A cc = Am 2 , cos 2% , (10) 

where Am]; = m 2 — mf is the mass squared difference between the two involved states and % is 
the corresponding mixing angle. For — > v s {y s is a sterile state) or u T — > v s resonant transition 
the condition is 

A NC = -Amjj cos 2% , (11) 
while for v e — > v s the resonance condition is 

A CC ~ A NC = Ami cos 2 % • ( 12 ) 

Since Acc and A^c are both positive for neutrinos and both negative for antineutrinos, and since 
Y e ~ 0.5 giving Acc — ^A^c for Earth matter, this resonance condition (12) is satisfied for 
neutrinos when Am 2 , > and for antineutrinos when Am 2 ^ < 0. On the other hand, the — > v s 
or v T — > v s resonance condition will be satisfied for neutrinos (antineutrinos) when Am]; < 
(Amjj > 0). The v e — > or v e — > v T resonance will happen in the neutrino (antineutrino) 
channel when Am 2 j > (Am 2 ^ < 0). Since we have both neutrinos and antineutrinos coming 
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from the atmosphere, we could have resonance in either the neutrino or the antineutrino channel 
for a given sgn(Am 2 j), if any of the above conditions are satisfied. 

All results presented in this paper are generated by exactly solving the full set of evolution 
equations for the neutrinos travelling through Earth matter parameterized by the PREM density 
profile [34]. However, just for simplicity let us for the moment assume that the neutrino travel 
through constant density matter inside the Earth. Assuming that Y e = 0.5, the very high energy 
atmospheric neutrinos travelling through the Earth would pick matter potential of 

A cc = 1.907 (eV 2 ) x ( 9 ) (^ttt) , (13) 

V5.0 gm/cc/ V5.0 TeV/ 

A NC = 0.954 (.VV^)^), (14) 
Acc - Anc = 0.953 (.V*) x ( 5 ^- 7 -) (^) . (15) 

Therefore, we see that the resonance conditions given by Eqs. (10), (11), and (12) can be easily 
satisfied for Am^ and Am^ relevant for LSND and MiniBooNE, where % = 1, 2, 3. We could also 
turn around these equations to give us the resonance energy as 

fsr-eiw) = Ami ;° B29 ' 4 . , no) 

0.076 x p(gm/cc) 

BST-(TeV) = Am '5 C ° B29 « , (17) 
0.038 x p(gm/cc) 

BSr-(TeV) = A '"'f B2fa , . (18) 
0.038 x p(gm/cc) 

where a in the above equations refer to either the /iorr flavor. For the Am| case we have the 
same relations with Am^ replaced by Am^. Therefore, for neutrinos crossing the core (mantle 
only) of the Earth for which the average matter density is ~ 8 gm/cc (~ 5 gm/cc), we expect 
resonance for E ~ 3 GeV (~ 5 GeV) if we assume Am^ and Am| to be about 1 eV 2 . We note 
that the energy at which we expect to see v e — > v s resonance is the same as the one where we are 
expecting to get v a — > v s resonance. On the other hand the energy at which we will get v e — > u a 
resonance will be lower by a factor of about 2. We get similar expressions also for Am^ induced 
resonances. Note that even though we have given the discussion for the Am 2 ~ eV 2 driven v e — > v a 
resonance for completeness, this resonance never happens in the 3+1 or 3+2 scenario inside the 
Earth since the Am 2 involved between the v e and u a states are the ones needed to explain the solar 
and atmospheric neutrino data and hence definitely not of the eV 2 scale that we are interested in. 
The mass eigenstates v± and v§ are predominantly composed of the sterile components and hence 
the Am 2 ! and Am^ mass squared difference drive the active-sterile resonances only. We remind 
the reader of the well known fact that when the resonance condition is satisfied, the corresponding 
mixing angle, even if it was very small in vacuum, becomes maximal in matter. Thus the amplitude 
factor in the oscillation probability also becomes maximal. 
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The oscillation probabilities in matter are given by the most general expression 



P fh (L)= 5 Pl -4E^Kf/ 7 T<^)sin 2(Am -' )ML 

i>i 



AE 



KufU^U^) sin ( -^f±, (19) 



where (Am? ) M and ?7 M are respectively the modified mass squared difference and mixing matrix 
in matter. The mixing matrix is parameterized in terms of the mixing angles (in matter). For the 
3+1 case we will have 6 mixing angles, while for the 3+2 scenario U M is given in terms of 10 angles. 
In fact, the most general form of the mixing matrix U M is complex and this CP dependence is 
probed through the last term in Eq. (19). But for simplicity, we will put all CP violating phases 
in U M to zero and hence the last term in Eq. (19) goes to zero. Also for TeV energy neutrinos, 
the oscillations induced by Ara^ and Am^ are negligible and we will get contributions from 
mainly the oscillatory terms corresponding to the mass squared difference associated with the 
sterile states. We reiterate that each term in the oscillation probability contains a product of 
the mass squared dependent oscillatory term and the mixing angle dependent term. Therefore, 
to achieve maximal oscillations in Earth matter, it is not enough to satisfy only the condition of 
resonance where the mixing angle becomes maximal. We should should simultaneously have the 
peak of the oscillatory term [35]. Thus one obtains maximal oscillations when the condition 

/x 33.55 x 10 3 
P L (km gm/cc) = — , (20) 

is satisfied. We have assumed a constant density for the Earth matter, L is the baseline where 
maximal oscillations happen and p is the corresponding average density. We will discuss this issue 
in greater details in the following section. 



3 Neutrino Oscillations with 3+1 Mass Spectrum 

We start by showing results for the case where there is only one extra sterile neutrino. For 4 
neutrinos we have 3 mass squared differences. For Am^ and Am^ we take the current best-fit 
values coming from global neutrino oscillation data, while for Am^ we take different values in the 
eV 2 range. For the mixing matrix we choose the following convention: 



u = R(e 34 )R(e 24 )R(e 23 )R(9 14 )R(e 13 )R(e 12 ) 



(21) 



where R(0ij) are the rotation matrices and 0^ the mixing angle. In general for the 3+1 scenario 
there are 3 CP violating Dirac phases. However as mentioned before, we have put all phases 
to zero in Eq. (21) for simplicity. The expressions for the oscillation probabilities relevant for 
atmospheric neutrinos then take the simple form 



sin 



2 <sm 



2 2< + cos 2 <sm 



2 2< 







j sin 2 


[ AE 



(22) 
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Figure 1: Values of sin 2 #24 at which we have maximal oscillations as a function of the distance L 
travelled inside Earth. We have assumed sin 2 # 14 = and sin 2 #34 = and the 3+1 mass spectrum. 
The dashed line shows the boundary between the mantle and core of the Earth. 
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AE 



• 2 nnM 2 nM • 2 /iM • 2 

sm cos v 2 \ sin fc^j sm 



r(Am| x ) 


M L~\ 


I AE 




[(Am 2 ,; 


M L~\ 


L 4£ 




[(Ami) 




L 4£ J 



(23) 
(24) 
(25) 
(26) 
(27) 



We note that the probabilities depend only on the 3 extra mixing angles #14, #24 and #34 . In par- 
ticular, P Me and P^ depend explictly on #14 and #24 and it seems that it is apparently independent 
of #34. We will see that this is not the case always and there is an implicit 6*34 dependence due to 
matter effects. 

The mixing angle #14 affects strongly the P ee channel. But in this section, we keep fixed #14 = 
for simplicity and concentrate on the oscillation channels affecting the muon type (anti) neutrino. 
We will probe the impact of #14 in the more realistic 3+2 scenario in next section. We first present 
results where the mixing angle #34 is also fixed at and only #24 is the non-zero sterile mixing 
angle. Finally we present results where both #24 and #34 are non-zero. 
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Figure 2: The survival probability P w as a function of L using the PREM profile for the Earth 
density. The different line types correspond to different fixed values of E and each panel shows 
the results for different fixed values of sin 2 #24 given in the panels. We have assumed the 3+1 mass 
spectrum and taken (Am 2 ^ = 1 eV 2 , sin 2 6*14 = and sin 2 ^ 34 = 0. The probability corresponds 
to neutrinos for Am?, < and to antineutrinos for Am?, > 0. 



3.1 Oscillation Probabilities in 3+1 when #14 = and #34 = 

For both # 14 = and #34 = 0, the probabilities assume very simple forms 



P„„ ~ l-sin 2 2<sin 2 



MM 



p ~ 
1 fie — 



(Am^) M L 



AE 



sin 2 29% sin 2 



o p ~ 

U ) J /IT 



4£ 

0, P fiT ~0, P f 



(28) 

(29) 
(30) 



This is therefore a case of simple two-generation — v s oscillations. The mixing angle and mass 
squared difference in matter are given as 



At/i 2 

sin 29$l = sin 20 24 11 



(Am 2 !) M ' 



(31) 



(Amir 



(A77& cos 2^4 ± v^GiriVApreP) 2 + (A77& sin2^) 2 



(32) 
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Figure 3: The survival probability P MAl as a function of L using the PREM profile for the Earth 
density. The different line types correspond to different fixed values of jAm 2 ^ and each panel 
shows the results for different fixed values of E given in the panels. We have assumed the 3+1 
mass spectrum and taken sin 2 ^24 = 0.04 eV 2 , sin 2 6*14 = and sin 2 6*34 = 0. The probability 
corresponds to neutrinos for Am| x < and to antineutrinos for Am 2 ^ > 0. 



where the + sign is for neutrino and — sign for the antineutrinos. Note that we have used 
Am^ ~ Am| 2 . As discussed before, we have resonant matter effects and sin 2 2^24 = 1 in the 
neutrino (antineutrino) channel when Am 2 x < (Am^ > 0). However, the condition of resonance 
does not necessarily give the largest possible oscillations. The condition for maximal oscillation 
is achieved when both sin 20|f = 1 and sin 2 [(Am 2 1 ) M L/4E] = 1 simultaneously [35] and is given 
by Eq. (20). It can be inverted to give the value of #24 which would give maximal oscillations at 
a given baseline: 

32.55 x 10 3 

tan 2fl 24 = -rj, ■ 33 

pL (km gm/ccj 

For values of #24 either less or greater than the value corresponding to that obtained from Eq. 
(33), the oscillations are less [36]. We show in Fig. 1 the value of sin 2 #24 for which we can 
have maximal oscillations, as a function of distance L travelled inside Earth. For p we have used 
the average density along the neutrino trajectory given by the PREM profile. The dashed line 
shows the mantle-core boundary of Earth and we can see that for the more plausible values of 
sin 2 #24 ^ 0.07 the condition for maximal oscillations are always met inside the Earth's core. In 
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particular, for the longest possible trajectory where L = 2 x Re, Re being the Earth's radius, we 
note that maximal oscillations will happen if sin 2 6* 2 4 = 0.02. 

Fig. 2 shows the survival probability P m as a function of the distance travelled inside the 
Earth. For this plot we use the full PREM density profile for the Earth and solve the four neutrino 
differential equation in matter. The different line types correspond to different fixed values of E 
and each panel shows the results for different fixed values of sin 2 6> 2 4 given in the panels. The 
probability shown would correspond to that for neutrinos if Am| x < and to antineutrinos if 
Am^j > 0. We note from the figure that for most neutrino energies above E ^ 2 TeV, there are 
sizeable matter effects inside Earth and the survival probability generally decreases with L in the 
mantle. Inside the core it falls first, followed by a rise. However, we can see that for reasonable L 
binning of the high energy atmospheric neutrino data in neutrino telescopes, it should be possible 
to see zenith angle dependent fall in P^. Even for very small values of sin 2 # 24 like 0.01, we can 
see that P w could fall to up to 0.6, and this should be observable in the detector. 

Fig. 3 is similar to Fig. 2, except that here we show the probability at a fixed value of sin 2 624, 
but different choices of [Amf-J and E. All plots are for sin 2 624 = 0.04 and each panel for a fixed 
E, shown in the figure. The different line types correspond to different jAm 2 ^. As before, the 
probability corresponds to neutrinos for Am^ < and to antineutrinos for Am^ > 0. This figure 
tells us how different values of lAmfJ can be distinguished from the high energy atmospheric 
neutrino data at the neutrino telescopes. We can see that binning in either or both E and L 
would help in distinguishing between the different possible (Amf^ values. 

3.2 Oscillation Probabilities in 3+1 when 6*14 = and 6*34 7^ 
If we allow sin 2 634 7^ but still keep 8 14 = 0, the probabilities are given as 



p 


~ 1 sin 2 29% sin 2 


AE \ ' 


(34) 


P 

1 fiT 


~ sin 2 2<sin 2 C sin 2 


"(Am^Ll 
AE J 


(35) 


P 

1 [IS 


~ sin 2 29% cos 2 9% sin 2 


r(Am 2 41 ) M L- 
AE 


(36) 


P 


~ 0, P er ~0, P ee - I. 


(37) 



Fig. 4 shows the neutrino oscillation probabilities assuming inverted mass ordering, taking Am^ = 
— 1 eV 2 . The — + v^, — > v T and — * v s probabilities are shown in the upper left hand panel, 
upper right hand panel and lower right hand panel, respectively. For sin 2 #14 = 0, the probabilities 
P^ e ~ 0, P eT — and P ee — 1 and hence we do not show them. The solid black (dark) line is for 
neutrinos in matter and the solid cyan (light) line is for antineutrinos in matter, while the thin 
black dashed line shows the probabilities in vacuum for comparison. We stress that even though 
we have denoted the probabilities as etc. in the figure, its understood that we are using 

them to denote the probability for both the neutrino as well as the antineutrino channels. We 
have kept sin 2 6> 2 4 = sin 2 9 U = 0.04 in this figure. The corresponding plots with the normal mass 
ordering is shown in Fig. 5. The neutrinos (antineutrinos) undergo maximal oscillations around 
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E (TeV) E (TeV) 

Figure 4: The — > (upper left hand panel), — > v T (upper right hand panel) and — > v a 
(lower right hand panel) oscillation probabilities, as a function of the neutrino energy E for the 3+1 
mass spectrum, when the neutrinos travel a distance L = 2R e , where R e is the radius of the Earth. 
The black (dark) solid lines show the probabilities for neutrinos while the cyan (light) solid lines 
show the probabilities for antineutrinos. The dashed lines give the probabilities if matter effects 
were not taken into account and one had oscillations in vacuum. The values of the oscillation 
parameters taken for this figure is shown in the lower left hand panel. In particular this plot is 
for Am^j = 1 eV 2 . Probabilities in matter have been obtained using the PREM profile. 



E = 2 TeV when Am^ = — 1 eV 2 (Am^ = +1 eV 2 ). At lower values of E, where very large 
matter effects in Am^ oscillations have still not set in, we note a marked difference between the 
oscillations pattern of neutrinos and antineutrinos and between the cases where Am 2 x < and 
Am^j > 0. The oscillations for lower E are dependent on Am 2 x as well and the difference between 
the oscillation mentioned above is due to both Am^ and Am| x dependent terms. Note that in 
both cases we have kept Am 2 x > 0. The most important thing to note from this figure is that 
around the point where we have maximal matter effects, ~ 0, P MT ~ 1 and P MS ~ for the 
neutrino (antineutrino) channel for Am 2 x < (Am 2 x > 0). Such large oscillations should not be 
difficult to observe in the very high atmospheric neutrino data in neutrino telescopes. 

In the previous subsection where we had put #34 = 0, we had argued that for L = 2R e , where 
R e is the Earth's radius, maximal oscillations of would occur around sin 2 #24 — 0.02. For 
sin 2 #24 = 0.04 we should therefore expect lesser oscillations. We had drawn these conclusions 
using average constant matter density approximation for the Earth. For the PREM profile which 
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corresponds to varying matter density for the Earth, this scenario holds, albeit approximately. 
In Figs. 4 and 5 we can see that we get maximal oscillations even for sin 2 # 2 4 = 0.04 as long as 
sin 2 #34 = 0.04. Another important aspect we note from the figures is that P /tr ~ 1 and P^ s ~ 
when we have maximal oscillations of v^. This is in stark contrast to the case where sin 2 # 34 = 0, 
for which we had P /tT ~ and P^ s ~ 1 (cf. Eqs. (29) and (30)). The main reason for this complete 
reversal of scenario is that when sin 2 #24 7^ and sin 2 #34 = 0, sin 2 O^i is enhanced in matter for 
neutrinos (antineutrinos) when Am^ < (Am| x > 0) while sin 2 remains zero. Therefore, P^ T 
is always zero and we have simple two-generation matter enhanced — > v a oscillations. However, 
when sin 2 #34 7^ 0, both sin 2 #24 and sin 2 #34 are enhanced in matter for neutrinos (antineutrinos) 
when Am 2 x < (Am 2 x > 0). This is a genuine three-generation oscillation case in which if 
sin 2 #34 = sin 2 #24, the and v T states evolve identically in matter and resonate with the sterile 
state at almost the same energy. 

To further illustrate this point we present Fig. 6, where we compare the probabilities corre- 
sponding to L = 2R e , for the cases where sin 2 #34 = with those where sin 2 #34 7^ 0. We show P w , 
P Mr and P^ s for sin 2 # 24 = 0.02, sin 2 6 34 = 0.00 (black solid lines), sin 2 # 24 = 0.02, sin 2 6 34 = 0.02 
(red dot-dashed lines), sin 2 6* 2 4 = 0.04, sin 2 #34 = 0.00 (green dotted lines), and sin 2 #24 = 0.04, 
sin 2 #34 = 0.02 (blue dashed lines). We reconfirm that for sin 2 #24 = 0.02 and sin 2 #34 = we have 
P MA1 ~ and P^s ~ 1. For sin 2 #24 = 0.04 and sin 2 #34 = 0, we still have two-generation — > u s os- 
cillations with P^ = 0, but now since we have shifted from the most optimal sin 2 #24 value for this 
baseline, ? w increases and P^ s decreases compared to the case where sin 2 #24 = 0.02. Once we put 
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E (TeV) E (TeV) 

Figure 6: The — > (upper left hand panel), — > u T (upper right hand panel) and — > v s 
(lower right hand panel) oscillation probabilities, as a function of the neutrino energy E, when 
the neutrinos travel a distance L = 2R e , where R e is the radius of the Earth. Different line types 
correspond to different combinations of sin 2 #24 and sin 2 #34 and Am^ = — 1 eV 2 . 



sin 2 #34 = sin 2 #24, we get P^ ~ for both sin 2 6*24 = 0.02 and sin 2 #24 = 0.04. However, non-zero 
sin 2 #34 brings a huge change in P^ T which becomes non-zero and large and for sin 2 #34 = 0.04, it 
is in fact very close to 1. Likewise, P^ s changes substantially due to sin 2 #34. 

In order to quantify our discussion on the impact of sin 2 #34 on the evolution of the neutrino 
states inside Earth, we show in Fig. 7 the mixing angles in matter as a function of E. Since its not 
possible to show the evolution of the mixing angles for the full PREM profile, we show a snapshot 
for a density of p = 8.44 gm/cc, which is the average density encountered by a neutrino moving 
along the diameter of the Earth. The green dot-dashed line shows sin 6™ when sin 2 9 2A = 0.04 
and sin 2 # 34 = in vacuum. We see that the mixing angle increases with energy and hence matter 
effects. At the resonance energy we get sin 2 29 2 f = 1 and beyond that sin 2 keeps increasing 
to 1 and sin 2 29^ decreases. The evolution of sin 2 9 2 { when sin 2 #24 = 0.04 and sin 2 #34 = 0.04 is 
shown by the black dotted line. This case is very different from the earlier described case. Here 
sin 2 #24 remains more or less constant beyond the resonance and hence sin 2 29 2 l assumes some 
large constant value and does not decrease like before. Note that the reason we were getting lesser 
oscillations in the P w channel for sin #24 = 0.04 (and sin 2 9 3A = 0) was because the resonance 
energy here was not matching exactly with the energy as which the oscillatory term peaks. But 
for sin 2 # 2 4 — 0.04 and sin 2 # 34 = 0.04 since sin 2 29%l has a large value from energies beyond 
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Figure 7: The mixing angles in matter as a function of the neutrino energy. The dotted and solid 
lines show sin 2 and sin 2 9^ when their values in vacuum are sin 2 # 2 4 = 0.04 and sin 2 # 34 = 0.04. 
The dotted-dashed lines show sin 9^i when the values in vacuum are sin 

#24 = 0.04 and sin 2 # 34 = 

0.00. 



the resonance, the problem of fine tuning the resonance energy and oscillations peak energy is 
drastically reduced and we can have P w ~ more easily. The reason for the change in behavior 
of P^it and P^ s with sin 2 # 34 ^ can also be seen from Fig. 7. The red solid line shows the 
angle sin 2 9^[ and we see that this keeps increasing and goes to 1 for E greater than the resonance 
energy. Since P^ T is proportional to sin 2 6^ and P^ s to cos 2 ^, P^ T increases while P^ s decreases 
as sin 2 9^ increases. 

4 Neutrino Oscillations with 3+2 Mass Spectrum 

The 3+1 neutrino mass spectrum, though simpler for the understanding of the resonant oscilla- 
tion picture, stands disfavored comprehensively once the latest MiniBooNE results are included 
into the analysis along with results from all other neutrino oscillation experiments. However, the 
3+2 scheme, with two extra sterile neutrinos, provides a very reasonable description of the world 
neutrino data, including MiniBooNE. In this section we look at the predicted neutrino and an- 
tineutrino oscillation probabilities at neutrino telescopes for the 3+2 neutrino mass scheme. The 
expression for the oscillation probabilities in the 3+2 scheme are as follows: 

^ = 1 - 4|Cff|'(l - Wtl?) sin 2 ( {Am !f L ) ~ 4|^!! 2 (l - IC/gl 2 ) Bin 2 L ) 
+ « Wl'sin {^MH) sin ((^jp) sin (<^) , ,38, 
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E (TeV) E (TeV) 



Figure 8: The — > (upper left hand panel), — > v T (upper right hand panel), — > v e (lower 
left hand panel), and v e — > v s \ and v e — > z/ S 2 (lower right hand panel) oscillation probabilities, 
as a function of the neutrino energy E for the 3+2 mass spectrum, when the neutrinos travel a 
distance L = 2R e , where R e is the radius of the Earth. The solid black and dashed blue lines 
show the probabilities for neutrinos while the solid cyan and thin dashed magenta lines are for the 
antineutrinos. The solid black and solid cyan lines are drawn for sin 2 #34 = sin 2 #35 = sin 2 #45 = 
0.01, while the dashed blue and dashed magenta lines are for sin 2 #34 = sin 2 #35 = sin 2 #45 = 0.0. 
The values of the other oscillation parameters are shown in the figure and we have taken Am^ = 
—0.87 eV 2 and Am^ = —1.91 eV 2 . For — > u s , we show the — ► v a \ by thick line and — > u s2 
by thin line. 



p - A\n M \ 2 \Tr M \ 2 ^-n 2 T ( Am 4i) L \ ±1ti m\2\ tt m\2- 2 f (A^hi) 

F a p— %^oA\ \ U ^\ Sm I )+*\ U ah\ \ U Po\ Sm I 

+ 8«^|^(^^)*(^^)cc.(^^) (39) 

Note that if we had taken into account CP violating phases in U, the argument in the cosine of 
the last term in Eq. (39) would be (Am 2 4 ) M L/4£ - 5 af3 where 5 aP = Arg(U&U$*U*£*U%). 

For the three active and two sterile neutrino framework we have 4 independent mass squared 
differences and hence can have the following possibilities for the mass spectrum which we call [14] : 

N2 + N3 : Am^ > 0, Am 2 x > and Am 2 51 > , (40) 
N2 + 13 : Am 2 x < 0, Am 2 x > and Am 2 51 > , (41) 
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E (TeV) E (TeV) 

Figure 9: Same as Fig. 8 but for Am^ = +0.87 eV 2 and Am^ = +1.91 eV 2 . For — > u s , we 
show the — > v s \ by thin line and — > z/ S 2 by thick line. 



H2 + N3(a) 
H2 + 13(a) 
I2 + N3 
12 + 13 



Amj! > 0, Am 2 x > and Am 2 x < 
Am 2 x < 0, 
Am 2 ! > 0, 



Am, 2 ! > and Am 2 x < , 
Am 2 x < and Am 2 , < , 



51 



Am 2 ! < 0, Am 2 ! < an d Am 2 x < 



(42) 
(43) 
(44) 
(45) 



with Am 2 ,! > always. In addition, the H2+N3 and H2+I3 schemes can have 2 more possibilities 
[15] 



H2 + N3(b) : Am 2 ! > °> Am 4i < and Am 5i > 



H2 + 13(b) : 



Am 2 ! 



<0, 



Am 2 ! 



< and Am 2 ,! > • 



(46) 
(47) 



Since there are two mass squared difference associated with the sterile states, we expect two 
resonances. Whether the resonance occurs in the neutrino or the antineutrino channel depends on 
the mass ordering. While the ordering of the mass states within the three active part is almost 
inconsequential for the very high energy neutrinos we are concerned with here, the mass ordering 
of the sterile states between themselves and with respect to the three active states is of utmost 
importance. In particular, if both Am^ > and Am 2 x > (corresponding to the N2+N3 and 
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Figure 10: The v e — > v e (upper left hand panel), v e — > u T (upper right hand panel), v e — > (lower 
left hand panel), and v e — + v s \ and v e — > v S 2 (lower right hand panel) oscillation probabilities, as a 
function of the neutrino energy E for the 3+2 mass spectrum, when the neutrinos travel a distance 
L = 2R e , where R e is the radius of the Earth. The solid black lines show the probabilities for 
neutrinos while the solid cyan lines are for the antineutrinos. For v e — > u s , we show the v e — > u sl 
by thick line and v e — > v s2 by thin line. We have taken Am^ = 0.87 eV 2 and Am^ = 1.91 eV 2 . 
The oscillation probabilities mainly depend on only sin 2 # 14 and sin 2 # 15 and almost independent 
of all other mixing angles. 



N2+I3 spectra), then both the v ^ — > v s [y e v s ) resonances happen in the antineutrino (neutrino) 
channel. On the other hand, if both Am^ < and Am^ < (corresponding to the I2+N3 and 
12+13 spectra), then both the — > v s [y e — > u s ) resonances happen in the neutrino (antineutrino) 
channel. For the hybrid cases, where Am 2 x > and Am 2 x < (H2+N3(a) and H2+I3(a)) or 
Am^ < and Am 2 x > (H2+N3(b) and H2+I3(b)), one of the resonances occur in the neutrino 
and another in the antineutrino channel. 

Figs. 8 and 9 show the probabilities involving the muon (anti)neutrino, as a function of energy 
for the I2+N3 and N2+N3 cases respectively. The solid black (dark) and dashed blue (dark) lines 
are for neutrinos and solid cyan (light) and dashed magenta (light) lines are for antineutrinos. We 
show results for the global best-fit parameter values taken from [12]. When the entire MiniBooNE 
data is included in the analysis, the authors of [12] get as their best-fit lAm 2 ^ = 0.87 eV 2 and 
lAmjy = 1.91 eV 2 for the mass squared difference and |[/ e4 | = 0.12, \U e5 \ = 0.11, jt/^l = 0.18 and 
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\U^5\ — 0.089. If we assume a parameterization for the 5x5 mixing matrix as 

u = R(e 45 )R(e 35 )R(e u )R(e 25 )R(e 24 )R(e 15 )R(e u )R(e 23 )R(e 13 )R(e 12 ) , (48) 

then the best-fit values for the matrix elements mentioned above can be obtained if we take 
sin 2 0i4 = 0.014, sin 2 15 = 0.012, sin 2 9 24 = 0.034 and sin 2 9 25 = 0.008. We present our results 
assuming these values. The other mixing angles associated with the sterile states are sin 2 34 , 
sin 2 6*35 and sin 2 45 . These remain almost unconstrained by the current neutrino oscillation data 
and could in principle take any value. For the sake of illustration, we show results only for two 
sets of choices for these mixing angles. The solid lines show probabilities for sin 2 #34 = sin 2 #35 = 
sin 2 #45 = 0.01, while the dashed lines are for sin 2 #34 = sin 2 9 35 = sin 2 # 45 = 0.0. The other 
mixing angles are fixed at sin 2 9 12 = 0.3, sin 2 9 23 = 0.5 and sin 2 13 = 0.01. We see that for the 
I2+N3 spectrum both resonances are in the neutrino channel while for the N2+I3 spectrum both 
resonances are in the antineutrino channel. We see that the effect of the mixing angles sin 2 # 34 and 
sin 2 #35 is to increase the — > v T oscillations and reduce the — > v s transitions in the resonant 
channel. The net result of these mixing angles is to reduce slightly the net survival probability. 
These features are similar to what we had observed for non-zero sin 2 #34 for the 3+1 case discussed 
in the previous section. The reason why P^ e is very small is easy to see from Eq. (39). In the 
neutrino channel, say, with Am| x < and Am 2 x < as in Fig. 8, while UM and Uj£ increase due 
to resonance, C/Jf and remain negligible since 9fl remains small, and as a result P Me remains 
negligible. In the antineutrino channel for this mass spectrum and are large (as discussed 
below), however in that case Uff 4 and Uf^ are small. Therefore, P Me is always small. Similarly, 
using Eq. (39) it is easy to see that P^ is large when sin 2 # 34 and sin 2 # 35 are large. In fact, its easy 
to see that non-zero sin 2 # 34 brings the Am^ driven first peak in P^ T and non-zero sin 2 #35 brings 
the Amjj driven second peak. Using similar arguments one can check that the muon neutrinos 
oscillate into the first sterile neutrino at the Am^ driven resonance and into the second sterile 
neutrino at the Am^ driven resonance. 

Since the mixing angle sin 2 #14 is non-zero, we expect resonant transitions for electron neutrinos 
as well. We show in Fig. 10 the oscillation probabilities associated with the electron type neutrinos 
and antineutrinos. The upper left hand panel shows the survival probability P ee , the upper right 
hand panel shows P er , the lower left hand panel shows P eM , while the lower right hand panel shows 
the transition probability to the first sterile state, P eSl and to the second sterile state, P eS2 . We 
have assumed the N2+N3 (or N2+I3) spectrum for the neutrinos with the current global best-fit 
numbers for the oscillation parameters. The black (dark) lines are for neutrinos while the cyan 
(light) lines are for antineutrinos. For the I2+N3 (and 12+13) spectra, the black (dark) lines 
would be for antineutrinos and cyan (light) lines for neutrinos. As discussed before, we see that 
both resonances come in the neutrino (antineutrino) channel when Ara^ > and Am^ > 
(A77& < and Ami^ < 0). For the hybrid cases H2+N3(a) and H2+N3(b) (as well as H2+I3(a) 
and H2+I3(b)), only one resonance will occur in either the neutrino or the antineutrino channel 
depending whether the mass squared difference is positive or negative respectively. 

The expressions for the probabilities involving the electron neutrino in the 3+2 picture using 
the parameterization for U given by Eq. (48) are 



P ee ~l - cos 4 9™ sin 2 29™ sin 2 



AE 
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(49) 



The mixing angles sin 2 9f[ and sin 2 9 M do not reach maximal value simultaneously. Therefore, we 
can see from this expression that we would have 2 big dips in the survival probability due to the 
first and the second terms when we have the Am 41 and Am^ driven resonances respectively. The 



last term is proportional to sin 9fl sin 29^ and needs 9fl and and 9^ to be large simultaneously. 
A third dip would be possible only when this condition is satisfied. The transition probabilities 
have the general form given by Eq. (39). The reason why P eil ~ and P eT ~ is same as 
that discussed before. If the v e — > v s resonance happens in the neutrino channel, the — > v s 
resonance will happen in the antineutrino channel. Therefore, when sin 2 9 M = 1 or sin 2 9 M = 1 
due to v e — > v s resonance, the other mixing angles do not receive any matter enhancement. One 
can check that in this case the mixing matrix elements UM, U^, I 
the sterile mixing angles in vacuum are small and we have P e „ ~ and P er ~ 



and U™ are very small if 



5 Flavor and Event Ratios with Sterile Neutrinos 

Neutrino telescopes, such as IceCube and Km3Net are not expected to have any charge identi- 
fication capability. Therefore, they will not be able to distinguish the neutrino signal from the 
antineutrino signal. This is particularly relevant for matter effects, since resonant transitions due 
to Earth matter is large in only either the neutrino or the antineutrino channel for a given sign 
of the mass squared difference which drives the resonance. Consequently, one should look at the 
sum of the neutrino and antineutrino events expected at the neutrino telescope. The threshold 
energies for muon, electron and tau detection in IceCube are about 100 GeV, 1 TeV and 1000 
TeV, respectively. It is easiest to see muon events, which leave distinct tracks in the detector. 
Both the electron events and the tau events produce showers. In principle the tau events should 
be separable from the electron events in certain energy range where they produce the so-called 
"double bang" signal in the detector [37]. However, because tau events have energy threshold of 
about 1000 TeV, they are not of any interest to us. Therefore, we could consider the simple event 
ratio 

r= t^t;> (50) 

where N a are the number of events observed corresponding to the species type a. The number of 
events is mainly given in terms of the (anti) neutrino flux, cross-section and the relevant oscillation 
probabilities. A detail prescription for calculating the number of events due to atmospheric is 
given in [26]. In this paper, we will not attempt to calculate the number of events exactly, which 
in addition to the main quantities mentioned above, also depend on other things like distance 
covered and energy loss of the lepton inside ice, details of the detector and the necessary cuts of 
the experiment. Instead, just for the purpose of illustration, we present the ratio of the product 
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Figure 11: Binned result for the ratio R as a function of energy E. The upper left panel shows the 
result for the zenith bin —0.2 > cos #2 > —0.4, the upper right panel for —0.4 > cos #2 > —0.6, the 
lower left panel for —0.6 > cos #2 > —0.8, and the lower right panel for —0.8 > cos #2 > —1.0. The 
solid lines show the expected R in the respective zenith bins when we have no sterile neutrinos 
and there are only three generation oscillations. The other 4 line types correspond to N2+N3 
(long dashed red lines), 12+13 (dot-dashed green lines), H2+N3(a) (dashed magenta lines) and 
H2+N3(b) (dotted blue lines). For all cases we have taken lAm^l = 0.87 eV 2 , (Am 2 ^ = 1.91 
eV 2 and mixing angles corresponding to their global best-fit values. We take sin 2 #34 = sin 2 #35 = 
sin 2 #45 = 0.01. 



of the flux, cross-section and the relevant probabilities as 

[4>u e Pee + + [4>u e Pee + (fiv^PfieWu 

We use the atmospheric neutrino flux given by Honda et. al. [38] and high energy charged current 
cross-sections from [39]. The efficiency of observing electron events is smaller than for muon 
events. However the difference in the 1 — 10 TeV range is small and therefore we neglect that 
here, since what we present is merely for illustration only. IceCube is expected to have rather 
good zenith angle resolution of about 25° [40] and in [26] the authors have presented their results 
in 5 energy bins between 1 — 10 TeV. In Fig. 11 we show the zenith angle binned value for R, as 
a function of the energy E. We have divided the zenith angle range —0.2 > cos# z > —1.0, into 
4 bins and show results where we have calculated R by summing over the product of the flux, 
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cross-section and relevant probabilities in the zenith bins. The upper left panel shows the result 
for the zenith bin —0.2 > cos# z > —0.4, the upper right panel for —0.4 > cos# 2 > —0.6, the lower 
left panel for —0.6 > cos 6^ > —0.8, and the lower right panel for —0.8 > cos9 z > —1.0. The solid 
black lines show the expected R in the respective zenith bins when we have no sterile neutrinos 
and there are only three generation oscillations. Note that these three generation oscillations are 
important for neutrino energies up to 1 TeV for neutrinos travelling large distances inside Earth. 
The other 4 line types correspond to the 4 relevant mass spectra discussed for the 3+2 scheme, 
N2+N3 (long dashed red lines), 12+13 (dot-dashed green lines), H2+N3(a) (dashed magenta lines) 
and H2+N3(b) (dotted blue lines). For all cases we have taken (Am^-J = 0.87 eV 2 , (Am 2 ^ = 1.91 
eV 2 and mixing angles corresponding to their global best-fit values, as discussed in Section 4. The 
hitherto unconstrained mixing angles sin 2 # 34 , sin 2 #35 and sin 2 # 45 are taken as 0.01. We see that 
there is a huge change in the value of R due to presence of sterile neutrinos for all values of E 
between 1 and 10 TeV. The change is also seen to be clearly dependent on the energy and zenith 
angle of the neutrinos. We note from the figure that not only should it be possible to establish 
the presence of sterile neutrinos from the observations, it should also be possible to differentiate 
between the different 3+2 neutrino mass because each one of them has a distinct prediction for R. 

Few comments are in order. The number of muon events expected from high energy atmo- 
spheric neutrinos has been given in Table 1 of [26], for 2 zenith angle bins —0.6 > cos 6^ > — 1.0 
and —0.2 > cos #2 > —0.6 and different energy bins. In the —0.6 > cos^ > —1.0 bin, about 
52,474 muon events are expected in 0.1 < E (TeV) < 0.16 energy bin, about 3,330 events in 
0.25 < E (TeV) < 3.98 energy bin and about 1,721 events in 3.98 < E (TeV) < 6.31 en- 
ergy bin, after 10 years of IceCube operation. Higher number of events are expected in the 
—0.2 > cos 9 Z > —0.6 zenith angle bin. Thus we expect a rather good statistic atmospheric neu- 
trino data at the neutrino telescopes. In fact, with such high statistic, it should be possible to look 
for sterile neutrinos in the atmospheric neutrino data sample using just the muon events alone. 

6 Conclusions 

Following the recently declared MiniBooNE results, sterile neutrinos have been the focus of dis- 
cussions in the field of neutrino physics. In particular, the question whether MiniBooNE data has 
unambiguously ruled out the possibility of sterile neutrinos needed to explain the LSND results 
has been raised. In this paper we have expounded the possibility of answering this question using 
the high energy atmospheric neutrino data in the upcoming neutrino telescopes. 

If sterile neutrinos exist with Am 2 ~ eV 2 , we expect to see flavor oscillations of upward going 
atmospheric neutrinos with the peak in the transition probability at an energy of a few TeV. One 
could naively think that these oscillations would normally be small owing to the smallness of the 
sterile mixing angles, which are severely constrained by the short baseline oscillation experiment 
data. We pointed out that near-resonant matter effects driven by the sterile neutrino mass squared 
differences drive these very small mixing angles in vacuum to almost maximal in matter. For a 
given neutrino baseline inside the Earth, the largest oscillations of course occur when the condition 
of resonance and the condition of oscillation peak are simultaneously satisfied. We showed in the 
framework of the simpler 3+1 neutrino mass scenario that this condition could be satisfied for TeV 
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neutrinos crossing the Earth. We assumed a simple framework where the only non-zero sterile 
angle was #24 and calculated the value of the mixing angle for which one could have maximal 
oscillations into sterile neutrino. For #24 both larger and smaller than this critical value, one would 
get lesser oscillation. We next allowed for non-zero 6*34 values and studied how this mixing angle 
changed the oscillation scenario. We showed that matter effects simultaneously enhance both 
#24 and #34 and we have genuine three generation effects in the oscillation probability. One very 
important effect is that with non-zero # 34 the P MT oscillation probability increases significantly and 
could even become maximal, while the P^ s probability simultaneously decreases. 

We considered the still viable 3+2 neutrino mass and mixing scheme and presented the oscil- 
lation probabilities when all mixing angles were allowed to be non-zero, as needed to explain the 
global oscillation data, including LSND and MiniBooNE. For the 3+2 mass scheme one can have 
as many as 8 different mass ordering. Of these, there are at least 4 different possibilities that 
would allow for resonant matter transition driven by the sterile mass eigenstates and each one 
gives a distinct signature in the oscillation pattern. We presented the results for the oscillation 
probabilities obtained by evolving the full five generation neutrino system inside the Earth matter 
as they travel, assuming the PREM profile for the matter density. We explained these results 
using simplified constant matter density picture. We showed that the mixing angles #34 and #35 
which are enhanced inside the Earth matter cause P^ T to increase significantly. We emphasized 
the fact that while — > v s resonance occurs for Am 2 < in the neutrino channel, the v e — > v s 
resonance condition is satisfied for Am 2 > in the neutrino channel. In the antineutrino channel 
of course the sign of Am 2 is reversed for the resonance condition to be satisfied. We showed how 
this feature ensured that the transition probability P efJi and P er always remained negligible. For 
#14 and/or #15 non-zero, we showed how resonance in the neutrino (antineutrino) channel produces 
huge dips in P ee . 

Finally, we discussed how these large matter effects due to presence of sterile neutrinos would 
show up in the neutrino telescopes. Atmospheric neutrinos form a "background" for the ultra high 
energy neutrino observation in the neutrino telescopes. These atmospheric neutrinos are in the 
TeV range where we expect near-resonant matter effects. At these energies hundreds of thousands 
of muon type events are expected from atmospheric neutrinos in IceCube and we argued that even 
a very moderate energy and zenith angle resolution in the data would lead to an unambiguous 
signal for sterile neutrinos. 

Note Added 

After the first version of the this paper appeared, the Super-Kamiokande collaboration have re- 
leased their data on showering muon events, which come from neutrinos with energies in the TeV 
range [41]. They comment on the feasibility of using their very high energy upward-going muon 
data sample which come from showing type events to shed light on the existence of high Am 2 
solutions, and cite this paper. 
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